{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0053e014",
   "metadata": {},
   "source": [
    "# Day 3：数据清洗与整理（保证“干净”）\n",
    "\n",
    "> 今日目标：能识别并处理缺失、重复、类型不一致问题；完成一个简单“清洗报告”。\n",
    "\n",
    "学习路径：\n",
    "1. 为什么清洗重要 (质量 → 可信分析)\n",
    "2. 读取脏数据并初步诊断\n",
    "3. 缺失值识别与处理 (dropna / fillna)\n",
    "4. 重复值处理 (duplicated / drop_duplicates)\n",
    "5. 类型转换 (to_datetime / astype)\n",
    "6. 条件过滤 (示例：PM2.5 > 阈值)\n",
    "7. Mini Case：生成“清洗前后对比”摘要\n",
    "8. 小结\n",
    "9. 课后作业提示"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12822fc6",
   "metadata": {},
   "source": [
    "## 1. 为什么数据清洗重要？\n",
    "- 脏数据会导致误判：平均值偏差、趋势失真\n",
    "- 可复现性需要记录清洗步骤\n",
    "- 真实世界数据常见问题：缺失/重复/格式不一/异常值\n",
    "\n",
    "思政点：公共数据治理中，高质量数据是科学决策与公平政策的基础。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42987e7c",
   "metadata": {},
   "source": [
    "## 2. 读取“脏”数据并初步诊断\n",
    "示例文件：`air_quality_dirty.csv` 包含：\n",
    "- 重复行 (同一日期+城市重复)\n",
    "- 缺失值 (空白 或 NA)\n",
    "- 日期格式不统一 (YYYY-MM-DD vs YYYY/MM/DD)\n",
    "- 数值缺失 (空 / NA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2f89f0d4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.microsoft.datawrangler.viewer.v0+json": {
       "columns": [
        {
         "name": "index",
         "rawType": "int64",
         "type": "integer"
        },
        {
         "name": "date",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "city",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "province",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "PM25",
         "rawType": "float64",
         "type": "float"
        },
        {
         "name": "PM10",
         "rawType": "float64",
         "type": "float"
        },
        {
         "name": "NO2",
         "rawType": "float64",
         "type": "float"
        },
        {
         "name": "SO2",
         "rawType": "int64",
         "type": "integer"
        }
       ],
       "ref": "d0391160-54a2-4b44-8032-4894366d9867",
       "rows": [
        [
         "0",
         "2025-09-01",
         "广州",
         "广东",
         "42.0",
         "55.0",
         "19.0",
         "7"
        ],
        [
         "1",
         "2025-09-01",
         "广州",
         "广东",
         "42.0",
         "55.0",
         "19.0",
         "7"
        ],
        [
         "2",
         "2025-09-01",
         "深圳",
         "广东",
         "35.0",
         "48.0",
         "17.0",
         "6"
        ],
        [
         "3",
         "2025-09-01",
         "佛山",
         "广东",
         null,
         "70.0",
         "23.0",
         "9"
        ],
        [
         "4",
         "2025-09-02",
         "北京",
         "北京",
         "60.0",
         "80.0",
         "35.0",
         "12"
        ],
        [
         "5",
         "2025-09-02",
         "上海",
         "上海",
         "45.0",
         "62.0",
         null,
         "10"
        ],
        [
         "6",
         "2025-09-02",
         "杭州",
         "浙江",
         "38.0",
         "50.0",
         "20.0",
         "7"
        ],
        [
         "7",
         "2025-09-02",
         "成都",
         "四川",
         "58.0",
         "75.0",
         "30.0",
         "11"
        ]
       ],
       "shape": {
        "columns": 7,
        "rows": 8
       }
      },
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>city</th>\n",
       "      <th>province</th>\n",
       "      <th>PM25</th>\n",
       "      <th>PM10</th>\n",
       "      <th>NO2</th>\n",
       "      <th>SO2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2025-09-01</td>\n",
       "      <td>广州</td>\n",
       "      <td>广东</td>\n",
       "      <td>42.0</td>\n",
       "      <td>55.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2025-09-01</td>\n",
       "      <td>广州</td>\n",
       "      <td>广东</td>\n",
       "      <td>42.0</td>\n",
       "      <td>55.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2025-09-01</td>\n",
       "      <td>深圳</td>\n",
       "      <td>广东</td>\n",
       "      <td>35.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2025-09-01</td>\n",
       "      <td>佛山</td>\n",
       "      <td>广东</td>\n",
       "      <td>NaN</td>\n",
       "      <td>70.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2025-09-02</td>\n",
       "      <td>北京</td>\n",
       "      <td>北京</td>\n",
       "      <td>60.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2025-09-02</td>\n",
       "      <td>上海</td>\n",
       "      <td>上海</td>\n",
       "      <td>45.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2025-09-02</td>\n",
       "      <td>杭州</td>\n",
       "      <td>浙江</td>\n",
       "      <td>38.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2025-09-02</td>\n",
       "      <td>成都</td>\n",
       "      <td>四川</td>\n",
       "      <td>58.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         date city province  PM25  PM10   NO2  SO2\n",
       "0  2025-09-01   广州       广东  42.0  55.0  19.0    7\n",
       "1  2025-09-01   广州       广东  42.0  55.0  19.0    7\n",
       "2  2025-09-01   深圳       广东  35.0  48.0  17.0    6\n",
       "3  2025-09-01   佛山       广东   NaN  70.0  23.0    9\n",
       "4  2025-09-02   北京       北京  60.0  80.0  35.0   12\n",
       "5  2025-09-02   上海       上海  45.0  62.0   NaN   10\n",
       "6  2025-09-02   杭州       浙江  38.0  50.0  20.0    7\n",
       "7  2025-09-02   成都       四川  58.0  75.0  30.0   11"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "dirty = pd.read_csv('../data/air_quality_dirty.csv')\n",
    "dirty.head(8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c276401d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "形状 shape: (14, 7)\n",
      "info():\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 14 entries, 0 to 13\n",
      "Data columns (total 7 columns):\n",
      " #   Column    Non-Null Count  Dtype  \n",
      "---  ------    --------------  -----  \n",
      " 0   date      14 non-null     object \n",
      " 1   city      14 non-null     object \n",
      " 2   province  14 non-null     object \n",
      " 3   PM25      12 non-null     float64\n",
      " 4   PM10      13 non-null     float64\n",
      " 5   NO2       13 non-null     float64\n",
      " 6   SO2       14 non-null     int64  \n",
      "dtypes: float64(3), int64(1), object(3)\n",
      "memory usage: 916.0+ bytes\n",
      "None\n",
      "各列缺失值计数:\n",
      "date        0\n",
      "city        0\n",
      "province    0\n",
      "PM25        2\n",
      "PM10        1\n",
      "NO2         1\n",
      "SO2         0\n",
      "dtype: int64\n"
     ]
    },
    {
     "data": {
      "application/vnd.microsoft.datawrangler.viewer.v0+json": {
       "columns": [
        {
         "name": "index",
         "rawType": "int64",
         "type": "integer"
        },
        {
         "name": "date",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "city",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "province",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "PM25",
         "rawType": "float64",
         "type": "float"
        },
        {
         "name": "PM10",
         "rawType": "float64",
         "type": "float"
        },
        {
         "name": "NO2",
         "rawType": "float64",
         "type": "float"
        },
        {
         "name": "SO2",
         "rawType": "int64",
         "type": "integer"
        }
       ],
       "ref": "4e9f62b7-1501-44be-b0b8-8bbeed3d815d",
       "rows": [
        [
         "9",
         "2025-09-03",
         "武汉",
         "湖北",
         "48.0",
         "65.0",
         "26.0",
         "9"
        ],
        [
         "10",
         "2025-09-03",
         "西安",
         "陕西",
         "63.0",
         "85.0",
         "37.0",
         "14"
        ],
        [
         "11",
         "2025/09/03",
         "西安",
         "陕西",
         "63.0",
         "85.0",
         "37.0",
         "14"
        ],
        [
         "12",
         "2025-09-03",
         "衡阳",
         "湖南",
         null,
         "68.0",
         "25.0",
         "8"
        ],
        [
         "13",
         "2025-09-03",
         "株洲",
         "湖南",
         "50.0",
         null,
         "24.0",
         "8"
        ]
       ],
       "shape": {
        "columns": 7,
        "rows": 5
       }
      },
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>city</th>\n",
       "      <th>province</th>\n",
       "      <th>PM25</th>\n",
       "      <th>PM10</th>\n",
       "      <th>NO2</th>\n",
       "      <th>SO2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2025-09-03</td>\n",
       "      <td>武汉</td>\n",
       "      <td>湖北</td>\n",
       "      <td>48.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2025-09-03</td>\n",
       "      <td>西安</td>\n",
       "      <td>陕西</td>\n",
       "      <td>63.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2025/09/03</td>\n",
       "      <td>西安</td>\n",
       "      <td>陕西</td>\n",
       "      <td>63.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2025-09-03</td>\n",
       "      <td>衡阳</td>\n",
       "      <td>湖南</td>\n",
       "      <td>NaN</td>\n",
       "      <td>68.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2025-09-03</td>\n",
       "      <td>株洲</td>\n",
       "      <td>湖南</td>\n",
       "      <td>50.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>24.0</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          date city province  PM25  PM10   NO2  SO2\n",
       "9   2025-09-03   武汉       湖北  48.0  65.0  26.0    9\n",
       "10  2025-09-03   西安       陕西  63.0  85.0  37.0   14\n",
       "11  2025/09/03   西安       陕西  63.0  85.0  37.0   14\n",
       "12  2025-09-03   衡阳       湖南   NaN  68.0  25.0    8\n",
       "13  2025-09-03   株洲       湖南  50.0   NaN  24.0    8"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 基本概况\n",
    "print('形状 shape:', dirty.shape)\n",
    "print('info():')\n",
    "print(dirty.info())\n",
    "\n",
    "print('各列缺失值计数:')\n",
    "print(dirty.isnull().sum())\n",
    "\n",
    "# 注意: 'NA' 字符串不是自动识别为缺失，需要指定或替换\n",
    "dirty.tail(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1591fbfe",
   "metadata": {},
   "source": [
    "### 小练习 1\n",
    "1. 使用 dirty.isnull().mean() 查看缺失占比。\n",
    "2. 思考：哪些列缺失会影响我们后续指标分析？\n",
    "(请自己在下方新建代码单元尝试)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7bb8584",
   "metadata": {},
   "source": [
    "## 3. 缺失值处理 (dropna / fillna)\n",
    "常见策略：\n",
    "- 删除行：dropna() (适合缺失少且不能合理填补)\n",
    "- 填充：fillna(均值/中位数/固定值)\n",
    "- 保留：标记缺失 (有时缺失本身有信息)\n",
    "\n",
    "这里演示：\n",
    "1. 将 'NA' 字符串替换为真正缺失 (np.nan)\n",
    "2. 分别计算删除法与填充值法的影响。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4e604469",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.microsoft.datawrangler.viewer.v0+json": {
       "columns": [
        {
         "name": "index",
         "rawType": "int64",
         "type": "integer"
        },
        {
         "name": "city",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "PM25",
         "rawType": "float64",
         "type": "float"
        },
        {
         "name": "PM10",
         "rawType": "float64",
         "type": "float"
        }
       ],
       "ref": "ce877f6a-6404-4fc7-a1f3-f0e654e8d3e2",
       "rows": [
        [
         "0",
         "广州",
         "42.0",
         "55.0"
        ],
        [
         "1",
         "广州",
         "42.0",
         "55.0"
        ],
        [
         "2",
         "深圳",
         "35.0",
         "48.0"
        ],
        [
         "3",
         "佛山",
         null,
         "70.0"
        ],
        [
         "4",
         "北京",
         "60.0",
         "80.0"
        ],
        [
         "5",
         "上海",
         "45.0",
         "62.0"
        ],
        [
         "6",
         "杭州",
         "38.0",
         "50.0"
        ],
        [
         "7",
         "成都",
         "58.0",
         "75.0"
        ],
        [
         "8",
         "重庆",
         "55.0",
         "72.0"
        ],
        [
         "9",
         "武汉",
         "48.0",
         "65.0"
        ]
       ],
       "shape": {
        "columns": 3,
        "rows": 10
       }
      },
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>city</th>\n",
       "      <th>PM25</th>\n",
       "      <th>PM10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>广州</td>\n",
       "      <td>42.0</td>\n",
       "      <td>55.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>广州</td>\n",
       "      <td>42.0</td>\n",
       "      <td>55.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>深圳</td>\n",
       "      <td>35.0</td>\n",
       "      <td>48.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>佛山</td>\n",
       "      <td>NaN</td>\n",
       "      <td>70.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>北京</td>\n",
       "      <td>60.0</td>\n",
       "      <td>80.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>上海</td>\n",
       "      <td>45.0</td>\n",
       "      <td>62.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>杭州</td>\n",
       "      <td>38.0</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>成都</td>\n",
       "      <td>58.0</td>\n",
       "      <td>75.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>重庆</td>\n",
       "      <td>55.0</td>\n",
       "      <td>72.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>武汉</td>\n",
       "      <td>48.0</td>\n",
       "      <td>65.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  city  PM25  PM10\n",
       "0   广州  42.0  55.0\n",
       "1   广州  42.0  55.0\n",
       "2   深圳  35.0  48.0\n",
       "3   佛山   NaN  70.0\n",
       "4   北京  60.0  80.0\n",
       "5   上海  45.0  62.0\n",
       "6   杭州  38.0  50.0\n",
       "7   成都  58.0  75.0\n",
       "8   重庆  55.0  72.0\n",
       "9   武汉  48.0  65.0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 替换 'NA' 为 np.nan\n",
    "dirty = dirty.replace('NA', np.nan)\n",
    "dirty[['city','PM25','PM10']].head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "04d5b6d8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "删除后形状: (10, 7)\n"
     ]
    },
    {
     "data": {
      "application/vnd.microsoft.datawrangler.viewer.v0+json": {
       "columns": [
        {
         "name": "index",
         "rawType": "int64",
         "type": "integer"
        },
        {
         "name": "PM25",
         "rawType": "float64",
         "type": "float"
        },
        {
         "name": "PM10",
         "rawType": "float64",
         "type": "float"
        }
       ],
       "ref": "66b204d8-5044-4a78-8080-fb93622b090f",
       "rows": [
        [
         "0",
         "42.0",
         "55.0"
        ],
        [
         "1",
         "42.0",
         "55.0"
        ],
        [
         "2",
         "35.0",
         "48.0"
        ],
        [
         "3",
         "49.916666666666664",
         "70.0"
        ],
        [
         "4",
         "60.0",
         "80.0"
        ]
       ],
       "shape": {
        "columns": 2,
        "rows": 5
       }
      },
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PM25</th>\n",
       "      <th>PM10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>42.000000</td>\n",
       "      <td>55.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>42.000000</td>\n",
       "      <td>55.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>35.000000</td>\n",
       "      <td>48.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49.916667</td>\n",
       "      <td>70.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>60.000000</td>\n",
       "      <td>80.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        PM25  PM10\n",
       "0  42.000000  55.0\n",
       "1  42.000000  55.0\n",
       "2  35.000000  48.0\n",
       "3  49.916667  70.0\n",
       "4  60.000000  80.0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 方案A：直接删除含任意缺失的行\n",
    "drop_any = dirty.dropna()\n",
    "print('删除后形状:', drop_any.shape)\n",
    "\n",
    "# 方案B：用列均值填充数值缺失\n",
    "fill_mean = dirty.copy()\n",
    "for col in ['PM25','PM10','NO2','SO2']:\n",
    "    fill_mean[col] = fill_mean[col].astype(float)\n",
    "    fill_mean[col] = fill_mean[col].fillna(fill_mean[col].mean())\n",
    "\n",
    "fill_mean[['PM25','PM10']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3d23bab3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "原始 PM25 均值: 49.92\n",
      "删除法 PM25 均值: 50.4\n",
      "均值填充后 PM25 均值: 49.92\n"
     ]
    }
   ],
   "source": [
    "# 对比填充前后 PM25 平均值变化\n",
    "orig_pm25_mean = dirty['PM25'].astype(float).mean()\n",
    "drop_pm25_mean = drop_any['PM25'].astype(float).mean()\n",
    "fill_pm25_mean = fill_mean['PM25'].astype(float).mean()\n",
    "print('原始 PM25 均值:', round(orig_pm25_mean,2))\n",
    "print('删除法 PM25 均值:', round(drop_pm25_mean,2))\n",
    "print('均值填充后 PM25 均值:', round(fill_pm25_mean,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fd3610e",
   "metadata": {},
   "source": [
    "### 小练习 2\n",
    "请尝试：\n",
    "1. 用中位数 (median) 填充 PM10 缺失，并比较均值变化。\n",
    "2. 思考：删除行是否可能引入偏差？为什么？"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cf8faef",
   "metadata": {},
   "source": [
    "## 4. 重复值处理 (duplicated / drop_duplicates)\n",
    "问题：第一行与第二行完全重复。\n",
    "\n",
    "流程：\n",
    "1. 使用 duplicated() 查看哪些行重复\n",
    "2. 计数后删除重复行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "18dd3e31",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "重复行数量: 1\n"
     ]
    },
    {
     "data": {
      "application/vnd.microsoft.datawrangler.viewer.v0+json": {
       "columns": [
        {
         "name": "index",
         "rawType": "int64",
         "type": "integer"
        },
        {
         "name": "date",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "city",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "province",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "PM25",
         "rawType": "float64",
         "type": "float"
        },
        {
         "name": "PM10",
         "rawType": "float64",
         "type": "float"
        },
        {
         "name": "NO2",
         "rawType": "float64",
         "type": "float"
        },
        {
         "name": "SO2",
         "rawType": "int64",
         "type": "integer"
        }
       ],
       "ref": "38f0eb82-7002-4673-84b6-9eb1b267333c",
       "rows": [
        [
         "1",
         "2025-09-01",
         "广州",
         "广东",
         "42.0",
         "55.0",
         "19.0",
         "7"
        ]
       ],
       "shape": {
        "columns": 7,
        "rows": 1
       }
      },
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>city</th>\n",
       "      <th>province</th>\n",
       "      <th>PM25</th>\n",
       "      <th>PM10</th>\n",
       "      <th>NO2</th>\n",
       "      <th>SO2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2025-09-01</td>\n",
       "      <td>广州</td>\n",
       "      <td>广东</td>\n",
       "      <td>42.0</td>\n",
       "      <td>55.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         date city province  PM25  PM10   NO2  SO2\n",
       "1  2025-09-01   广州       广东  42.0  55.0  19.0    7"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 标记重复行 (除第一出现外的重复)\n",
    "dup_mask = dirty.duplicated()\n",
    "print('重复行数量:', dup_mask.sum())\n",
    "dirty[dup_mask].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a090ced0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "删除重复后行数: 13\n"
     ]
    }
   ],
   "source": [
    "# 删除重复\n",
    "no_dup = dirty.drop_duplicates()\n",
    "print('删除重复后行数:', no_dup.shape[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7707ce1",
   "metadata": {},
   "source": [
    "### 小练习 3\n",
    "如果只判断 (date, city) 组合是否重复，应该如何写？提示：subset 参数。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6c779d1",
   "metadata": {},
   "source": [
    "## 5. 类型转换 (to_datetime / astype)\n",
    "问题：日期列存在两种格式 '2025-09-03' 与 '2025/09/03'。\n",
    "使用 to_datetime 统一，并观察失败情况 (errors='coerce')。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "26121f28",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.microsoft.datawrangler.viewer.v0+json": {
       "columns": [
        {
         "name": "index",
         "rawType": "int64",
         "type": "integer"
        },
        {
         "name": "date",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "date_parsed",
         "rawType": "datetime64[ns]",
         "type": "datetime"
        }
       ],
       "ref": "88eb0a25-8539-447f-8e3e-90ead64d3662",
       "rows": [
        [
         "0",
         "2025-09-01",
         "2025-09-01 00:00:00"
        ],
        [
         "1",
         "2025-09-01",
         "2025-09-01 00:00:00"
        ],
        [
         "2",
         "2025-09-01",
         "2025-09-01 00:00:00"
        ],
        [
         "3",
         "2025-09-01",
         "2025-09-01 00:00:00"
        ],
        [
         "4",
         "2025-09-02",
         "2025-09-02 00:00:00"
        ],
        [
         "5",
         "2025-09-02",
         "2025-09-02 00:00:00"
        ],
        [
         "6",
         "2025-09-02",
         "2025-09-02 00:00:00"
        ],
        [
         "7",
         "2025-09-02",
         "2025-09-02 00:00:00"
        ],
        [
         "8",
         "2025-09-03",
         "2025-09-03 00:00:00"
        ],
        [
         "9",
         "2025-09-03",
         "2025-09-03 00:00:00"
        ],
        [
         "10",
         "2025-09-03",
         "2025-09-03 00:00:00"
        ],
        [
         "11",
         "2025/09/03",
         null
        ]
       ],
       "shape": {
        "columns": 2,
        "rows": 12
       }
      },
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>date_parsed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2025-09-01</td>\n",
       "      <td>2025-09-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2025-09-01</td>\n",
       "      <td>2025-09-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2025-09-01</td>\n",
       "      <td>2025-09-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2025-09-01</td>\n",
       "      <td>2025-09-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2025-09-02</td>\n",
       "      <td>2025-09-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2025-09-02</td>\n",
       "      <td>2025-09-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2025-09-02</td>\n",
       "      <td>2025-09-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2025-09-02</td>\n",
       "      <td>2025-09-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2025-09-03</td>\n",
       "      <td>2025-09-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2025-09-03</td>\n",
       "      <td>2025-09-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2025-09-03</td>\n",
       "      <td>2025-09-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2025/09/03</td>\n",
       "      <td>NaT</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          date date_parsed\n",
       "0   2025-09-01  2025-09-01\n",
       "1   2025-09-01  2025-09-01\n",
       "2   2025-09-01  2025-09-01\n",
       "3   2025-09-01  2025-09-01\n",
       "4   2025-09-02  2025-09-02\n",
       "5   2025-09-02  2025-09-02\n",
       "6   2025-09-02  2025-09-02\n",
       "7   2025-09-02  2025-09-02\n",
       "8   2025-09-03  2025-09-03\n",
       "9   2025-09-03  2025-09-03\n",
       "10  2025-09-03  2025-09-03\n",
       "11  2025/09/03         NaT"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 尝试直接转换\n",
    "dirty['date_parsed'] = pd.to_datetime(dirty['date'], errors='coerce')\n",
    "dirty[['date','date_parsed']].head(12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "692b0f2f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "无法解析的日期数量: 1\n"
     ]
    }
   ],
   "source": [
    "# 查看无法解析的情况数量 (NaT 视为缺失)\n",
    "print('无法解析的日期数量:', dirty['date_parsed'].isnull().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae9f7a97",
   "metadata": {},
   "source": [
    "如果部分无法解析，可：\n",
    "1. 手动替换格式\n",
    "2. 指定 format 参数 (当格式统一时)\n",
    "当前示例已全部成功转换。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86e77e49",
   "metadata": {},
   "source": [
    "### 小练习 4\n",
    "1. 将 PM25 列强制转为 float (astype(float)) 并统计缺失。\n",
    "2. 统计每个城市记录条数 (value_counts)。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cde19e2",
   "metadata": {},
   "source": [
    "## 6. 条件过滤示例\n",
    "需求：筛选 PM25 > 55 的记录，关注可能污染较高的情况。\n",
    "写法：df[df['PM25'] > 55]。注意需先保证 PM25 为数值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6532069d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.microsoft.datawrangler.viewer.v0+json": {
       "columns": [
        {
         "name": "index",
         "rawType": "int64",
         "type": "integer"
        },
        {
         "name": "date",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "city",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "PM25",
         "rawType": "float64",
         "type": "float"
        }
       ],
       "ref": "fad39fa2-a6f5-4f1b-9b81-f5b965d2e270",
       "rows": [
        [
         "4",
         "2025-09-02",
         "北京",
         "60.0"
        ],
        [
         "7",
         "2025-09-02",
         "成都",
         "58.0"
        ],
        [
         "10",
         "2025-09-03",
         "西安",
         "63.0"
        ],
        [
         "11",
         "2025/09/03",
         "西安",
         "63.0"
        ]
       ],
       "shape": {
        "columns": 3,
        "rows": 4
       }
      },
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>city</th>\n",
       "      <th>PM25</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2025-09-02</td>\n",
       "      <td>北京</td>\n",
       "      <td>60.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2025-09-02</td>\n",
       "      <td>成都</td>\n",
       "      <td>58.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2025-09-03</td>\n",
       "      <td>西安</td>\n",
       "      <td>63.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2025/09/03</td>\n",
       "      <td>西安</td>\n",
       "      <td>63.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          date city  PM25\n",
       "4   2025-09-02   北京  60.0\n",
       "7   2025-09-02   成都  58.0\n",
       "10  2025-09-03   西安  63.0\n",
       "11  2025/09/03   西安  63.0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 确保数值型\n",
    "dirty['PM25'] = dirty['PM25'].astype(float)\n",
    "high_pm25 = dirty[dirty['PM25'] > 55]\n",
    "high_pm25[['date','city','PM25']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49577b23",
   "metadata": {},
   "source": [
    "拓展：多条件组合示例\n",
    "- (dirty['PM25'] > 55) & (dirty['province'] == '陕西')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d0a44ca",
   "metadata": {},
   "source": [
    "### 小练习 5\n",
    "1. 筛选 province == '广东' 且 PM10 > 60 的行。\n",
    "2. 思考：为什么加括号？(运算符优先级)。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85b2fc89",
   "metadata": {},
   "source": [
    "## 7. Mini Case：生成“清洗前后对比”摘要\n",
    "目标：输出一个简单字典或表，展示清洗效果：\n",
    "- 原始行数 vs 去重行数 vs 删除缺失后行数\n",
    "- PM25 原始均值 / 缺失填充后均值 (或删除后均值)\n",
    "- 缺失值总数变化\n",
    "\n",
    "下面给出示例代码骨架，可自行补充。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bd1381a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'原始行数': 14,\n",
       " '去重后行数': 13,\n",
       " '再删除缺失后行数': 9,\n",
       " '原始缺失总数': 4,\n",
       " '清洗后缺失总数': 0,\n",
       " 'PM25 原始均值': 49.92,\n",
       " 'PM25 删除缺失后均值': 51.33}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 重新读取，按顺序执行清洗，生成对比摘要\n",
    "dirty2 = pd.read_csv('../data/air_quality_dirty.csv').replace('NA', np.nan)\n",
    "orig_rows = dirty2.shape[0]\n",
    "orig_missing = dirty2.isnull().sum().sum()\n",
    "\n",
    "# 去重\n",
    "no_dup2 = dirty2.drop_duplicates()\n",
    "rows_no_dup = no_dup2.shape[0]\n",
    "\n",
    "# 删除含缺失行\n",
    "drop_na2 = no_dup2.dropna()\n",
    "rows_drop_na = drop_na2.shape[0]\n",
    "missing_after = drop_na2.isnull().sum().sum()\n",
    "\n",
    "# 均值对比 (仅演示 PM25)\n",
    "pm25_mean_orig = pd.to_numeric(dirty2['PM25'], errors='coerce').mean()\n",
    "pm25_mean_drop = pd.to_numeric(drop_na2['PM25'], errors='coerce').mean()\n",
    "\n",
    "summary = {\n",
    "    '原始行数': orig_rows,\n",
    "    '去重后行数': rows_no_dup,\n",
    "    '再删除缺失后行数': rows_drop_na,\n",
    "    '原始缺失总数': int(orig_missing),\n",
    "    '清洗后缺失总数': int(missing_after),\n",
    "    'PM25 原始均值': round(pm25_mean_orig, 2),\n",
    "    'PM25 删除缺失后均值': round(pm25_mean_drop, 2)\n",
    "}\n",
    "summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83e93f53",
   "metadata": {},
   "source": [
    "思考：如果用“填充”而不是“删除”，均值会向哪个方向偏？为什么？"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50c325cb",
   "metadata": {},
   "source": [
    "## 8. 小结\n",
    "\n",
    "今日关键点回顾：\n",
    "1. 缺失值：检测 isnull() / 处理 dropna() & fillna()；比较不同策略对均值的影响。\n",
    "2. 重复值：duplicated() / drop_duplicates()；可用 subset 精准指定判重字段。\n",
    "3. 类型转换：to_datetime 统一日期；astype 强制列类型。\n",
    "4. 条件过滤：布尔索引 (可多条件 & / | 组合，注意括号)。\n",
    "5. 清洗摘要：用“行数/缺失/均值变化”量化清洗效果，形成可复现记录。\n",
    "\n",
    "实践建议：\n",
    "- 任何替换与填充动作前，先保留一份原始数据副本。\n",
    "- 记录每一步理由（为何删除 / 填充 / 转换），方便复盘与说明可靠性。\n",
    "\n",
    "衔接提示：有了干净数据与清洗日志，Day 4 的探索统计才能避免“垃圾进 → 垃圾出”。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0739cd19",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PM10 异常行数: 0\n",
      "按 (date,city) 重复行数: 1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'step': 'drop_duplicates', 'rows_before': 14, 'rows_after': 13},\n",
       " {'step': 'dropna_all', 'rows_before': 13, 'rows_after': 13}]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 进阶清洗代码示例\n",
    "import pandas as pd, numpy as np\n",
    "dirty_demo = pd.read_csv('../data/air_quality_dirty.csv').replace('NA', np.nan)\n",
    "\n",
    "# 1. 分组填充示例（PM25 按城市均值填充）\n",
    "filled_group = dirty_demo.copy()\n",
    "filled_group['PM25'] = filled_group.groupby('city')['PM25'].transform(lambda s: pd.to_numeric(s, errors='coerce').fillna(pd.to_numeric(s, errors='coerce').mean()))\n",
    "\n",
    "# 2. IQR 简易异常标记 (对 PM10)\n",
    "pm10 = pd.to_numeric(filled_group['PM10'], errors='coerce')\n",
    "Q1, Q3 = pm10.quantile(0.25), pm10.quantile(0.75)\n",
    "IQR = Q3 - Q1\n",
    "upper = Q3 + 1.5 * IQR\n",
    "filled_group['PM10_outlier'] = pm10 > upper\n",
    "print('PM10 异常行数:', filled_group['PM10_outlier'].sum())\n",
    "\n",
    "# 3. 主键重复检查\n",
    "dup_key = filled_group.duplicated(subset=['date','city']).sum()\n",
    "print('按 (date,city) 重复行数:', dup_key)\n",
    "\n",
    "# 4. 清洗日志简单实现\n",
    "log = []\n",
    "def log_step(name, df_before, df_after):\n",
    "    log.append({'step': name, 'rows_before': len(df_before), 'rows_after': len(df_after)})\n",
    "\n",
    "# 示例：去重 + 删除全空行\n",
    "df0 = filled_group\n",
    "df1 = df0.drop_duplicates(); log_step('drop_duplicates', df0, df1)\n",
    "df2 = df1.dropna(how='all'); log_step('dropna_all', df1, df2)\n",
    "log"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b99230a9",
   "metadata": {},
   "source": [
    "### 清洗进阶与边界案例\n",
    "1. 连续缺失填充：时间序列可用 fillna(method='ffill') / bfill。\n",
    "2. 分组填充：按城市分别用各自均值填充：df.groupby('city')['PM25'].transform(lambda s: s.fillna(s.mean()))。\n",
    "3. 条件填充：仅对 PM25 缺失且同列 PM10 不缺失时填固定值。\n",
    "4. 异常检测雏形：利用 describe 的上下四分位距(IQR)，标记 > Q3 + 1.5*IQR 的值（了解）。\n",
    "5. 类型批量转换：select_dtypes(include='object') 遍历尝试 to_datetime / to_numeric(errors='ignore')。\n",
    "6. 字符串标准化：str.strip()/lower()/replace() 防止隐藏空格导致重复。\n",
    "7. 重复主键校验：期望 (date, city) 唯一，可用 duplicated(subset=['date','city']).any() 检查。\n",
    "8. 自定义“清洗日志”结构：append {'step':'dropna subset=...', 'rows_before':..,'rows_after':..}。\n",
    "9. inplace 争议：推荐结果重新赋值，减少副作用。\n",
    "10. 验证：清洗后做一次核心指标对比（均值/缺失率/行数）写表格存档。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c194b9a3",
   "metadata": {},
   "source": [
    "## 9. 课后作业提示\n",
    "在 *homework_day3.ipynb* 中：\n",
    "1. 复制今日 dirty 数据，新增 1 行自己设计的含缺失记录。\n",
    "2. 分别使用 删除法 / 均值填充法 处理缺失，比较某一列 (自选：PM10 或 NO2) 均值差异。\n",
    "3. 使用 subset 指定 (date, city) 去重并统计影响的行数。\n",
    "4. 筛选出 PM2.5 > 全部均值 的记录，统计有多少城市。\n",
    "5. 写 80~120 字：说明你认为“删除”和“填充”各自适用的场景。\n",
    "(可选) 6. 把清洗过程中的关键数字整理成一张 1 行摘要表。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85af39c5",
   "metadata": {},
   "source": [
    "---\n",
    "📌 提示：掌握布尔过滤与缺失处理，将直接影响 Day 4 的探索效率。继续保持‘每步记录’的习惯。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
